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Under National Assessrrent, several volumes of data have al- 
ready been collected and published. There no longer Is a debate 
concernlnR the pros and cons of having a National Assessment pro* 
gram, for the results of the early assessments are now available 
to the public, Edjcatnrs need to study and Interpret the results 
available as a base for decision making in education. Along with 
this I they need to understand how these data were collected and 
what were the probleiis and limitations of the data gathering 
process. 

National Assessment Is a plan for the systematic, census-like 
survey of knowledges, skills, understandings, and attitudes. It 
Is an Information gathering plan aimed at providing both educators 
and the lay public with Information concerning the level of achieve- 
ment in selected subject areas for students and young adults. The 
goal Is to provide Information that will be used to Improve educa- 
tion. It Is concerned with the achievement status of four age 
levels in ten different subject areas, Tl^ie subject areas selected 
for ttssessnent were: Art, Career and Occupational Development, 
Citizenship, Literature, Mathematics, Music, Reading, Science, 
Social Studies, and Writing, The basic task of this paper is to 
descrioe the model used by National Assessment for data gathering 
and reporting on the Citizenship area. It Is not within the scope 



Ut ri#*ll1M|l«T OP NtALTM. 
■ OUCATIOM a WtLPAIll 
M*T(OH*L (HITITUYt OP 
• OUCATION 

tMiS 00''<JMfKT MAS lltrN «f»»BO 
OUfltf tllAtTlV AS MfCflvrt) » WOV 
iHf Pf «SON r)»» OWf.AN'/*TKlN OM.Olhl 
ATiMf.it POINTS 0» vr 0« OPINIONS 
S^AirO DO NOT Nf f f SSAftiL V HfPHf 
'.I 'f "Al N/» T hONAl *N^T I T UTf O* 

I OW AT ON Pn'.l TiON OM P<n ^ 



of thl? paper to criticize the model with respect to technical 
flaws; hence^ the model Is de<;crlbed and discussed without rer 
feronce to any potential problems of design. 

THE MODEL 

The model Is in the continuous process of being refined and 
improved, thus only the basic components of the model are pre- 
sented in Diagram !• A circular scheme is used in presenting it. 
since, In reality, its actual application may be initiated with 
any one of the compor.ents. Also, In Its actual application, there 
are continual interactions between and among the various components. 
While theoretically the process starts with the refinement of over- 
all national goals into specific subject matter, behavioral ob- 
jectives, and pr.'^gresses in logical sequence through to the final 
Utilization of the Information, in practice there is much greater 
freedom with respect to the utilization of the components. 

The -model for the Citizenship Assessment is presented here in 
outline form with a fairly detailed description of its components. 
As presented In Llagram I, there are seven basic components identi- 
fied in the model: Objectlven Development, Exercises Development, 
Sampling Plan, Administration of Exercises, Scoring and Analysis, 
Reporting and Dlss-^minatlon, and Utilization of Information. While 
many of the fine uoints of the model are not developed in the fol- 
lowing outline, ^.t is described in sufficient detail to give the 
reader a good understanding of how the data were collected and what 
Implications night result from these data. The number of sub- 
topics in the model and their distribution indicate that the ma.lor 
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efforts of National Assessment have been with the first five com- 
ponents. The last three components have been areas of controversy 
and, therefore, have received less attention until recently, 

1 

Outline of ,Y\e Assessment Model for Cltl zenshto 
T, Objectives Development ^ ^ ^ 

A, The task of developing objectives In the field of 
citizenship was awarded to the American Institute for 
Research of Palo Alto, California, These criteria 
were used in examining the objectives: 

(a) They were considered important by scholars. 

(b) They were accepted as an educational task by the 
school. 

(c) They were considered desirable by thoughtful lay 
citizens. 

Scholars reviewed the objectives for authenticity 
with respect to their subject fields; school people 
reviewed the objectives in terras of their actual 
emphasis in their schools; and laymen reviewed them 
in terms of their experiences with regard to their 
value 3n life. 

B. The American Institute for Research staff reviewed 
previous lists of citizenship objectives and boiled 
these down to one comprehensive list of 20 objectives* 
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C. Outstanding local teachers familiar with each tar«»et- 
age group (P, 13, 17, adult), workin.-^ with the American 
Institute for Research staff, broke down each general 
objective into the post germane behaviors deemed appro- 
priate as goals for a given age group. 

D. A selected group of students and adults In each age 
group was asked by the American Institute for Research 
staff to recall and describe outstandint^ citizens of 
their acquaintance and specific incidents reflecting 
good and poor citizenship. These incide.-.ts and descrip- 
tions, about 1,000, were used to check the completeness 
of the initial list of objectives. 

E. The objectives were stated on three levels (general 
objectives, sub-objectives, and behavioral age illustra- 
tions or statements). The results were summarized for 
each age group. 

F. The revised list of objectives, broken down into Im- 
portant behaviors, was then worked over for three days 
by a panel of national leaders in citizenship education 
and related social sciences. 

G. A group of persons in various roles from selected 
California communities reviewed the objectives and made 
suggestions. These included public and private school 
administrators, counselors, teachers, a Judge, a county 
planner, labor and business leaders, and social 
scientists . 

H The objectives were then reviewed by panels of laymen. 
Eleven' lay review panels representing four geographic 
areas of the country and three different community sizes 
were used. Each panel spent two days reviewing the 
objectives based on these two questions: 'Is this some- 
thing important for people to learn today? ' and 
this something I would like to have my children l^arn? 

6 7 8 

II. Exercises Development ^ > 
^Womer^ od» cH. 

7Gadway, Charles J. (Ed.) Reading ^aM I>l tera ture : G|Mr|l 

Information Yearbook . Education Commission of the States, 
Report 02-Giy, i^ational Assessment of Educational Progress, 
Denver, Colo., May, 1^72. 

^Flnlev Carmen J. and Frances S. Berdie, The National Assess- 
^^"^ llWlS^Lch to mrcise Developnenl National Assessnent 
■[^Educational Progress, Ann Arbor, Mich., 19/0. 
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A The prorluction of the exercises was Initiated by the 
Atrerican Institute for Research in 1966. The exercises 
wfre developed to cover all of the ma^or ob.lectlves ^nd 
to represent the selected content areas. Many exercises 
required the use of Interview techniques, as well as the 
usual oencil and paper exercises. Also, self report and 
group task exercises were used. 

B. Because National Assessment intends to describe what 

people in an age group know, the exercises were written . 
to reflect three difficulty levels— reporting knowledge 
or skills common to almost all persons in an age group, 
reporting skills or understandings of a typical member 
of an age grouo, and reporting understandings or knowledge 
developed by the most able persons in an age group. 

C All exercises were developed to meet these criteria: 
content validity, clarity, functional exercise format, 
clustering exercises based on a single set of stimulus 
materials, directionality of response, difficulty level, 
content sampling, and overlap between age groups. The 
exercises were direct measures of some pieces of knowl- 
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dge, understandings, attitudes, or skills which were 
entioned in one or more of the objectives. 



D. The exercises were reviewed by panels of lay persons for 
clarity, neaningf ulness , and invasion of privacy. 

E There was a tryout of the exercises involving repre- 
* sentatives of groups in the actual assessment--regions , 
communities, races, sexes, and age groups. Following 
the tryouts the American Institute for Research staff 
and subject-matter specialists reviewed the tryout data 
and made needed revisions. 

F. A comnixtee of subject matter specialists "measurement 
special sts, and National Assessment staff members rated 
the excicises to be included in the packages according 
to a set of criteria, and based on the ratings the 
exercises were selected for use. 

o The selected exercises were reviewed by Office of 

Station personnel for any i"fri"^tSi"'o??eSs JenLs" 
the part of the respondents or possible of fensiveness. 

H <^inPP there were about l60 minutes of testing time 

av"laMe"foreach age group in each f.^J.f -^^nTlal 
exercises used were only a small sample of the potential 
number of exercises. The exercises were f f,^^"^^^?/"^^ 
administrative units (packages) for groups up to 12 
persons. 
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III. Sampling Plan ' 

A. The sampllnf: plnr- was subcontracted to Research Triangle 
Institute, HalGi/:h, North Carolina. A multi-sta^^e 
de-sign was used which was stratified by region, size of 
coT.munitv, and socio-economic status. This was a proba- 
bility sample which allowed researchers to collect data 
from a small sample of the population and to infer from 
that sample certain characteristics of the entire popula- 
tion. 

B. The populations for assessment were all 0 year olds, all 
13 year olds, all 17 year olds, and all younR adults 2b 
through 3^ years old in the 50 States plus the District 
of Coiurrbia. The only exceptions were the exclusions of 
institutionalized individuals of these ages— those in • • 
hospitals, prisons, and others who could not be reached. 

C. For apes 9 and 13, a school sample only was used and for 
the 26 through 35 a^e ^roup a household sample only was 
used. For the 17 year olds, both a school and a house- 
hold sample were used. 

D. The entire country was divided into population areas as 
follows: cities, counties exclusive of cities, and 
pseudo-counties~two or more counties were put together 
when the populrtlon of a single county was less than 
16,000. Each population unit of 16,000 residents was 
assigned a nunber. 

E. The country also was divided Into four geographic re- 
gions: Northeast, Southeast, Central, and V/est. 

F. Each geographic region was divided into communities of 
four tyt,es: large cities of above 180,000 population, 
urban fi-lnge, middle-sized cities between 25,000 to 
180,000 population, and small town-rr.ral of under 25,000 
population. 

G. The 52 sampling units for each geographic area were 
spread across the four community types in a fashion 
proportional to their population In relation to the 
area population. 

9worrls, Eleanor L., et al, Eej^t 1, 12i2-l^^ Helenas: - National 
Results ana lUnstr ations of Group Comparisons,^ J .^H.^Chrony 

and D. G 
Appendix 
Assessme 
1970. 

lOKorrls, Citizenship Objectives, on. ci^.. 
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H. To insure comparable "eprer-cntation from each part of 
the -country, an enual number of samplinc; units was 
selected from each neopraphic rep;; on — ^2 from each of 
the four repiions for i,. total of 208, 

T, The actual samplirift units chosen were selected at 

random. This plan did not guarantee that all ^0 States 
would be included in the sample. This was not a survey 
ob.^ective, but later the desipn was changed, so each 
state was included in the sample. 

J. In each sampling unit selected, all school buildings 
enrollin£» students of the sample ages (public, private, 
and parochial) were identified. 

K. The plan for schools was to select units of approximately 
250 to 350 pupils fcr each age group and from at least 
two different buildings within each sampling unit for 
each age group. 

L. Each cooperating building principal provided a list of 
names of students in the building from the specific age 
groups. This list was used for the final random selection 
of students to take the assessment exercises from that 
building. 

M. Information about the areas was obtained from the U. S, 
census lata. In order to report reliable information 
for lower socio-economic status groups, these groups 
were oversampled. There was a disproportionate number 
of schools from lower socio-economic status areas in- 
cluded. In the overall results, the data from the lower 
socio-economic areas were given the percentage value in 
which they occurred in the total population. 

N. From each of the 208 geographical samples, 100 adults, 
ages 26 through 35, were randomly selected using the 
rollowlug procedures. Each of the 208 geographic samples 
was divided into equal secondary sampling units. Then 
ten secondary sampling units were randomly selected from 
the total 208 samples. Interviewers then personally 
contacted the people in the chosen secondary sampling 
units of the 26 through 35 age group, and out-of-school 
17 year olds. These persons were asked to participate 
in the assessment. 

0. Individuals were classified as black, white, and other 
on the basis of information provided by the school or 
by observation. Results were given for black and white 
only. The number of individuals classified as other 
was too small to produce reliable results. 

11 12 

IV. Administration of. Exercises » 



^H/oner, o£» 

dway, jEiE. cit . 
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A. The administration of the exoiciser, was subcontracted 
to Research Trianfjlc Inj?titu;e in the Kast and to 
Measurement Reseairch Center ')f '-/Gstinpihouse Learning 
Corporation, Iowa City, Iowa, in the I7est. Cooperation 
of schools was obtained by first contacting officials 
at the state and then at school district levels. There 
was above 90 per cent cooperation by schools. 

Adults and out-of-school 17 year olds were contacted 
by a personal door-to-door household canvass. Each 
out-of-school participant was contacted individually. 
All had the right to refuse to cooperate. 

B. A full-time trained staff of 27 district supervisors 
managed the field work. They were assigned to different 
geographical areas of the United States. They contacted 
schools and recruited and trained local teachers to 
help in the administration of the exercises in schools 
and recruited and trained other available persons for 
the out-of-school administration. 

C. In the schools, students from a single age group from 
different classes were brought together in a room for 
exercise administration. Group size was at least 8 
and usually 12 students. 

D. The exercises were organized in packages which contained 
exercises from two or three different sub.lect areas at 

a single age level. No one person took all the exercises 
in his age group. Age groups were assessed at different 
times of the year. 

E. In packages administered to groups, taped directions and 
taped readings of the exercises were used in addition 

to printed packages. This was Aone to establish con- 
sistency in timing and administration plus to provide 
for nonreaders. 

P. Several Packages at ages 13, and 17 consisted of 

exercises that were given by exercise adminis waters to 
one individual at a time. The administration of all the 
packages for the adult assessment was done by interviews. 

G Each package required about 50 minutes of administrative 
time. Each person took only one package with the ex- 
ception of the out-of-school, 17 year olds who were 
asked to take four or five packages each since they were 
the most difficult and expensive group. to locate. 

H. Students' names were confidential and did not appear 

on any packages. The name roster was kept at the build- 
ing level and used only in the organization of the In- 
school sampling. 
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V, ScorinR and Analysis 
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A. The scorlnK and analysis of the exercises were sub- 
contracted to ><easurement Hosearch Center of Wes titighouno 
Learninf^ Corporation, Iowa City, Iowa. 

B. The multiple-choice exercises were scored and recorded 
routinely by machine. 

C. The openended exercises were scored by trained profes- 
sionals using a key of acceptable and unacceptable 
achievements in terms of the objectives. 

D. Results were reported for each goal. Also, the results 
were reported both as the percentage of any group of 
respondents making the desired responses to an exercise 
and as the difference between the percentage of a group 
making the desired responses and the corresponding 
national percentage.. 

E. In the assessment, there was a lack of proportionality 
among characteristics used in the comparison of groups, 
such as color, sex, parental education. A statistical 
procedure, balancing, was used to correct for this 
problem in the comparative analysis of the data. Balanc- 
ing is a procedure to examine the performance of groups 
classified on one characteristic adjusting for the fact 
that these groups differ on a s^iecified set of other 
characteristics . 

VI. Reporting and Dissemination-^^'^^'^^ 

A. The reporting of results was directed to subject matter 
specialists, professional educators, and informed lay- 
men, Multiple reports were dev loped to serve these 
different audiences. 

B. Approximately ^0 per cent of the exercises were reported 
at the end of each assessment year. Not all exercises 
were reported since they were to be used over again in 
future assessments in order to measure change by means 

of comparing the results on the uncontaminated exercises. 

C. The exercises released for publication were selected to 
be representative of all exercises administered as well 
as the results received on the assessment. 



13womer, o£. cit . 
I'+Gadway, o£. cit . 
l5Womer, oj), cit . 

I6canpbell, Peoort 2, CUU^en^-hi : N?^tion^l ]^e suits ,, on. cil. 
17Gadway, ofi. pit . 



11 



D. ' neoortinf^ was done by Q, 13, 17,. and 26 through 35 afie 

groups. Since the same exercises were used with dif- 
ferent aP.e Rroups, there was comnarable data across two 
or more ap.? levels. 

E. Reporting was also done by piroups within the categories 
of regions, community types, sex, socio-economic status, 
and white, black, and other. 

F. Final reports were printed with a short description of 
the exercises, the national percentage of success, and 
group differences from the national percentage of 
success for each exercise. This was done without any 
interpretation of results. 

G. Both observed and balanc«'d results for all exercises 
and by groups were reported. The effects of balancing 
on measured characteristics such as sex and region were 
included In the report. 

H. There were no scores reported for individuals. No 
single individual took more than one twelfth of the 
exercises, and no individual took a package that sampled 
only a single subject area. 

I. Results were reported through the media: written word, 
radio, television, films, and personal reports. 

VII. Utilization of Inf ormation^^»^9 

A. The results provided information for educational decision 
making. For example, considering the somewhat lower 
performance of the Southeast Region cn the Citizenship 
results, school boards in that region mi^jht decide to 
put greater stress In their school programs on citizen- 
ship skills, understandings, and attitudes. «iO 



l^V/omer, 02. cit . 

19conwav. l^irry E., "Some Implication-, of the National Assessment 

M^del and Data for State and Local lucation." Paper Presented 
at the 1073 Annual Meeting of the , 'A, New Orleans, Louisiana, 
February 26, 1Q73. 

20campbell, Vincent N., Manford J. Ferris, and Daryl G Nichols, 
National As sessment Report 6, lo6q-lQ22 Citizenship; QLOm 
Fp s ., 1 ts iEl ■LpJS t --^c ^ ^ Size, of Comrriunity, Education 
ct^JiitlTofthe-Stitil , National Assessment of Educational 
Progress, Denver, Colo., July, 1^71. 
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Tir The results raised many questions which may lead to 
other lnvestlp:atlons. For example, In making compari- 
sons of all CltlzenshlD results combined, It was found 
that the Kxtrene Affluent Suburbs showed substantial 
median advantages at all ages and that the Extreme Rural 
and Extreme Inner City showed substantial deficits at 
all ages. Here are discrepancies In performance which 
need to have causal studies conducted on them from the 
perspectives of different disciplines such as political 
science, sociology, economics, and education. 21 A 
second example is that assessment could expand Into 
other educational areas, such as assess a sample of 5 
year olds for a basis of comparison, break results down 
by states, or expand Into nev/ subject areas. 

C. The results of several cycles should provide evidence 
of the change In knowledge, skills, understandings, and 
attitudes In the age groups as they relate to educational 
objectives . 

D. School administrators can make comparisons between 
groups, and may Improve student performance from the 
information gained in this manner. 

From this review of the model, it is evident that the National 

Assessment staff has put a great deal of effort and know-how into 

the design, plus the development of each of the components. Each 

. of the components reflects the best available information in the 

field of assessment. 



SUMl^ARY 



A complicated but logical process has evolved for Identifying 
the behavioral objectives for the specific subject areas. The use 
of laymen along with teachers and subject matter specialists added 
validity checks on thesr objectives. Considerable Inventive talent 



was displayed in writing the exercises for assessing some of the 
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noncognltlvo typo of objectives. The use of the tape-paced 
administration was a valuable innovation in the field of testing. 
In this procedure, a conplote set of direc';iong for the admini- 
stration of a package of exercises Is recorded on an audio tape. 
By playing the tape, the directions are always given in the same 
manner and the exercises are tlrr.ed. Also, the problem of nonreaders 
is overcome through the oral directions. While the technology had 
been available prior to this use, it had not been widely used. The 
procedures for scoring subjective exercises have provided a model 
for accurate, objective scoring of these kinds of exercises. The 
scorers were put through a special training program for scoring 
essays, written materials, or performance exercises. 

The results have been released in census-like fashion without 
interpretation. Clear, factual reports were made of the data so 
the reader would know what was assessed and what were the results. 
Very little interpretation of the results has been done for the 
reader. This was the intent of the National Assessment staff, for 
they did not conceive of their role as going beyond the reporting 
function. 

The final two components of the model are the most contro- 
versial, for originally there was not a clear mandate for National 
Assessment to take a vigorous leadership role with respect to 
these two tasks. As time has gone by, greater activity by National 
Assessment in these areas has been urged by such groups as edu- 
cators and parents, and National Assessment has become more active 
in the dissemination and utilization components. 
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In r>umroary, it can be pointed out that the nodel reflects 
some Important choicer on the part of the staff of National 
Assessment. The decision was nade to assess a broad ranp:e of 
Db.iectlves In each of the ten subject matter fields. Certainly, 
it would have been much easier and cheaper to have concentrated 
on a narrow set of objectives. Also, the effort was successfully 
made to include the higher cognitive levels in the assessment 
exercises and to deal with the affective domain^ In subject areas 
like Citizenship and Social Studies, the usual standardized test 
concentrates on factual knowledge which is of a less controversial 
nature and easier to assess. National Assessment should be com- 
mended for their bolder, more comprehensive approach to the task 
which searches for more significant kinds of data^ 
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